--- title: Recreating Steffl's Flatfields keywords: fastai sidebar: home_sidebar summary: "Based on his thesis appendix" description: "Based on his thesis appendix" nb_path: "notebooks/06_calib.steffl.ipynb" ---
{% raw %}
{% endraw %} {% raw %}
{% endraw %}

missing = [] there = [] for id in obsids: try: data = UVPDS(id, skip_download=True) except FileNotFoundError: print(id, "not there.") missing.append(id) else: print("Got", id) there.append(id)

{% raw %} 
['2001-093', '2002-198', '2003-139']
{% endraw %} {% raw %} 
cat = CatalogFilter(steffl_spica_dates[2])

Stored index is up-to-date.
{% endraw %} {% raw %} 
pids = list(cat.get_euv_date().query("OBSERVATION_TYPE=='CALIB'").index)
{% endraw %} {% raw %} 
pids

['EUV2003_139_18_11',
 'EUV2003_139_19_07',
 'EUV2003_139_20_03',
 'EUV2003_139_21_00',
 'EUV2003_139_21_56',
 'EUV2003_139_22_52',
 'EUV2003_139_23_48']
{% endraw %} {% raw %} 
cat.set_next_day()
{% endraw %} {% raw %} 
pids.extend(list(cat.get_euv_date().query("OBSERVATION_TYPE=='CALIB'").index))
{% endraw %} {% raw %} 
pids

['EUV2003_139_18_11',
 'EUV2003_139_19_07',
 'EUV2003_139_20_03',
 'EUV2003_139_21_00',
 'EUV2003_139_21_56',
 'EUV2003_139_22_52',
 'EUV2003_139_23_48',
 'EUV2003_140_00_44',
 'EUV2003_140_01_41',
 'EUV2003_140_02_37',
 'EUV2003_140_03_33',
 'EUV2003_140_04_29',
 'EUV2003_140_05_25',
 'EUV2003_140_06_22']
{% endraw %} {% raw %} 
kwargs = {"x": "nx", "y": "ny", "cmap": "viridis", "clim": (0, 50)}
{% endraw %}

Row2Row correction

{% raw %} 
class Row2Row:
    def __init__(self, pid):
        self.pid = pid
        self.data = UVPDS(pid).xarray.astype("int16")

    @property
    def plot_set(self):
        return self.data.hvplot(
            x="spectral", y="spatial", cmap="viridis", title=self.pid
        )

    @property
    def integrated(self):
        return self.data.sum(dim="samples")

    @property
    def plot_integrated(self):
        return self.integrated.hvplot(
            x="spectral", y="spatial", cmap="viridis", title=self.pid
        )

    @property
    def averaged(self):
        return self.integrated.sel(spatial=slice(3, 61)).mean(dim="spatial")

    @property
    def plot_averaged(self):
        return self.averaged.hvplot(x="spectral", title=self.pid)

    @property
    def column_std(self):
        return self.integrated.sel(spatial=slice(2, 60)).std(dim="spatial")

    @property
    def plot_column_std(self):
        return self.column_std.hvplot(x="spectral", title=f"{self.pid}, Column STD")

    @property
    def ff(self):
        data = self.averaged / self.integrated
        data = data.where(np.isfinite(data), other=np.nan)
        data.loc[:, 61:] = 1
        data.loc[:, :3] = 1
        return data

    @property
    def plot_ff(self):
        return self.ff.hvplot(
            x="spectral", y="spatial", cmap="viridis", title=self.pid, clim=(None, 4)
        )
{% endraw %} {% raw %} 
pids

['EUV2003_139_18_11',
 'EUV2003_139_19_07',
 'EUV2003_139_20_03',
 'EUV2003_139_21_00',
 'EUV2003_139_21_56',
 'EUV2003_139_22_52',
 'EUV2003_139_23_48',
 'EUV2003_140_00_44',
 'EUV2003_140_01_41',
 'EUV2003_140_02_37',
 'EUV2003_140_03_33',
 'EUV2003_140_04_29',
 'EUV2003_140_05_25',
 'EUV2003_140_06_22']
{% endraw %} {% raw %} 
r2r = Row2Row(pids[0])
{% endraw %} {% raw %} 
r2r.plot_ff
{% endraw %} {% raw %} 
r2r.plot_integrated
{% endraw %} {% raw %} 
r2r.plot_column_std
{% endraw %} {% raw %} 
r2r.integrated.hvplot(x="spatial")
{% endraw %} {% raw %} 
r2r.ff

<xarray.DataArray 'EUV2003_139_18_11' (spectral: 1024, spatial: 64)>
1.0 1.0 1.0 1.0 0.9534 1.907 1.271 ... 1.266 1.266 1.055 0.7034 1.0 1.0 1.0
Coordinates:
  * spectral  (spectral) float64 111.5 111.6 111.7 111.7 ... 189.8 189.9 190.0
  * spatial   (spatial) int64 0 1 2 3 4 5 6 7 8 9 ... 55 56 57 58 59 60 61 62 63
Attributes:
    units:                 Counts
    long_name:             EUV raw data
    n_bands:               1024
    integration_duration:  Quantity(value=332.0, units='SECOND')
{% endraw %}

Col2Col sensitivity variation.

Universal detector stack xarray creator

This function allows to create detector-shaped stacks of data for different purposes, like:

  • along_slit scans
  • across_slit scans
  • sets of corrections for the Steffl flatfield
{% raw %} 
def create_detector_stack(
    data,  # numpy 3D array to embed to xarray
    name,  # name of data
    third_dim,  # name of third dimension ("scan", "along", "across")
    orig,  # xarray to copy coords from
):
    """Function to create a 2D detector array with variable 3rd dimensions.

    It uses an `orig` array to copy the coordinates from.
    """
    return xr.DataArray(
        data,
        dims=["spectral", "spatial"] + [third_dim],
        coords={
            "spectral": orig.spectral,
            "spatial": orig.spatial,
            third_dim: range(data.shape[-1]),
        },
        name=name,
    )
{% endraw %} {% raw %}

class Col2Col[source]

Col2Col(pids, i=15, m=0, plot_width=600) :: Parameterized

params(i=Integer, m=Integer, pids=List, name=String)

Parameters of 'Col2Col'

 Parameters changed from their default values are marked in red. Soft bound values are marked in cyan. C/V= Constant/Variable, RO/RW = ReadOnly/ReadWrite, AN=Allow None

Name Value Type Bounds Mode 

pids [] List (0, None) V RW i 0 Integer (0, 1023) V RW m 0 Integer (0, 13) V RW

Parameter docstrings: =====================

pids: List of product IDs i: Spectral column i m: Scan number m

Parameters:

  • pids : <class 'inspect._empty'>

    group of product ids for a raster run

  • i : <class 'int'>, optional

    Minimum column value for evaluation (15:997)

  • m : <class 'int'>, optional

    Default start scan

  • plot_width : <class 'int'>, optional
{% endraw %} {% raw %}
{% endraw %} {% raw %} 
c2c = Col2Col(pids)
{% endraw %} {% raw %} 
c2c.calculate_simple_correction()
{% endraw %} {% raw %} 
c2c.plot_all_Fm()
{% endraw %}
  • It is neat to see how the grid has a weaker signal in the last array, where only 2 columns could be combined
{% raw %} 
c2c.plot_simple_correction()
{% endraw %}

Row2Row flats

Every across_slit scan comes with an inner along_slit scan that can be used to create a row2row flatfield correction.

So we have 14 of them in this case, that can be averaged.

{% raw %} 
c2c.plot_r2r_flats()
{% endraw %} {% raw %} 
c2c.r2r_flat_average_plot()
{% endraw %}
  • Nice to see how the SNR below 155 nm has improved drastically from having the average of the 14 FF corrections.
{% raw %} 
c2c.simple_both_plot()
{% endraw %} {% raw %} 
c2c.i=100
c2c.m=0
{% endraw %} {% raw %} 
c2c.plot_triplet()
{% endraw %} {% raw %} 
c2c.plot_triplet(corrected=True)
{% endraw %} {% raw %} 
p1 = (
    c2c.arr.where(c2c.arr > 0, np.nan)
    .isel(across_slit=0, drop=True)
    .hvplot.image(x="spectral", cmap="viridis", logz=True, label="Original")
)
{% endraw %} {% raw %} 
dataset = c2c.arr.isel(across_slit=0, drop=True).to_dataset(name=c2c.arr.name)
{% endraw %} {% raw %} 
dataset

<xarray.Dataset>
Dimensions:   (spectral: 1024, spatial: 64)
Coordinates:
  * spectral  (spectral) float64 111.5 111.6 111.7 111.7 ... 189.8 189.9 190.0
  * spatial   (spatial) int64 0 1 2 3 4 5 6 7 8 9 ... 55 56 57 58 59 60 61 62 63
Data variables:
    Counts    (spectral, spatial) int64 0 0 4 5 4 2 3 3 ... 8 10 10 12 18 13 3 0
{% endraw %} {% raw %} 
dims = [c for c in dataset.coords if dataset[c].shape != ()][::-1]
dims

['spatial', 'spectral']
{% endraw %} {% raw %} 
index_dims = [d for d in dims if d in dataset.indexes]
index_dims

['spatial', 'spectral']
{% endraw %} {% raw %} 
c2c.arr.isel(across_slit=0, drop=True).indexes

spectral: Float64Index([             111.5, 111.57673509286413, 111.65347018572825,
                        111.73020527859238,  111.8069403714565, 111.88367546432063,
                        111.96041055718474, 112.03714565004887,   112.113880742913,
                        112.19061583577712,
                        ...
                        189.30938416422288,   189.386119257087, 189.46285434995113,
                        189.53958944281527, 189.61632453567938,  189.6930596285435,
                         189.7697947214076, 189.84652981427175,  189.9232649071359,
                                     190.0],
                       dtype='float64', name='spectral', length=1024)
spatial: Int64Index([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
                     17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
                     34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
                     51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63],
                    dtype='int64', name='spatial')
{% endraw %} {% raw %} 
p1
{% endraw %} {% raw %} 
p2 = (
    c2c.corrected_arr.where(c2c.corrected_arr > 0, np.nan)
    .isel(across_slit=0, drop=True)
    .hvplot.image(x="spectral", cmap="viridis", logz=True, label="Corrected")
)
{% endraw %} {% raw %} 
(p1+p2).cols(1)
{% endraw %}

Mean of inner area is 1:

{% raw %} 
c2c.simple_both.isel(spectral=slice(15, 998), spatial=slice(3,61)).mean().data

array(1.)
{% endraw %} {% raw %} 
iarr = c2c.arr.interactive()
{% endraw %} {% raw %} 
from panel import widgets
{% endraw %} {% raw %} 
ival = widgets.IntSlider(start=0, end=1024)
mval = widgets.IntSlider()
{% endraw %} {% raw %} 
iarr.isel(spectral=ival, drop=True).hvplot().layout()
{% endraw %} {% raw %} 
c2c.calculate_simple_correction()
{% endraw %} {% raw %} 
c2c.simple_correction

<xarray.DataArray 'simple col2col corr' (spectral: 1024, spatial: 64)>
-9.223e+18 -7.379e+18 0.6 1.0 1.8 ... 1.0 1.0 1.0 -1.845e+18 -7.379e+18
Coordinates:
  * spectral  (spectral) float64 111.5 111.6 111.7 111.7 ... 189.8 189.9 190.0
  * spatial   (spatial) int64 0 1 2 3 4 5 6 7 8 9 ... 55 56 57 58 59 60 61 62 63
{% endraw %} {% raw %} 
c2c.i = 200
c2c.m = 0
{% endraw %} {% raw %} 
c2c.column_set_mean() / c2c.column_set()[0]

<xarray.DataArray 'scan_stack' (spatial: 64)>
nan nan 0.875 0.9383 0.9184 0.8942 0.8806 ... 1.131 1.023 1.171 0.8434 2.583 nan
Coordinates:
  * spatial   (spatial) int64 0 1 2 3 4 5 6 7 8 9 ... 55 56 57 58 59 60 61 62 63
    scan      int64 0
    spectral  float64 126.8
{% endraw %} {% raw %} 
c2c.column_set()[0]

<xarray.DataArray 'scan_stack' (spatial: 64)>
0 0 2 18 9 16 8 13 13 12 17 11 10 10 17 ... 13 17 8 11 17 9 6 11 16 24 13 4 3 1
Coordinates:
    scan      int64 4
    spectral  float64 190.0
  * spatial   (spatial) int64 0 1 2 3 4 5 6 7 8 9 ... 55 56 57 58 59 60 61 62 63
{% endraw %} {% raw %} 
c2c.i = 200
{% endraw %} {% raw %} 
c2c.m = 3
{% endraw %} {% raw %} 
c2c.plot_averaged_triplet()
{% endraw %} {% raw %} 
c2c.plot_triplet(corrected=False, i=300)
{% endraw %} {% raw %} 
c2c.plot_simple_correction()
{% endraw %} {% raw %} 
c2c.simple_correction.shape

(1024, 64)
{% endraw %} {% raw %} 
c2c.plot_Fm(0)
{% endraw %} {% raw %} 
c2c.plot_triplet()
{% endraw %} {% raw %} 
c2c.column_set_mean()

array([0.00000000e+00, 0.00000000e+00, 1.03666667e+03, 2.44133333e+03,
       2.29500000e+03, 1.98833333e+03, 1.78433333e+03, 2.38500000e+03,
       2.24466667e+03, 1.91800000e+03, 2.22333333e+03, 2.17300000e+03,
       1.81233333e+03, 1.92266667e+03, 2.09233333e+03, 2.54666667e+03,
       2.03866667e+03, 2.56266667e+03, 2.21300000e+03, 1.91233333e+03,
       2.12633333e+03, 1.71133333e+03, 2.68400000e+03, 1.84933333e+03,
       2.63100000e+03, 2.06766667e+03, 2.64400000e+03, 1.92766667e+03,
       2.36400000e+03, 1.99966667e+03, 1.75333333e+03, 2.37766667e+03,
       2.27800000e+03, 1.64400000e+03, 2.56766667e+03, 2.60333333e+03,
       1.71633333e+03, 2.01800000e+03, 2.49433333e+03, 1.96033333e+03,
       2.62033333e+03, 2.24366667e+03, 2.15033333e+03, 2.63866667e+03,
       2.43900000e+03, 2.57533333e+03, 2.66033333e+03, 1.85633333e+03,
       1.98533333e+03, 2.56100000e+03, 2.50233333e+03, 2.69100000e+03,
       2.25366667e+03, 2.29566667e+03, 2.81633333e+03, 2.42533333e+03,
       1.98266667e+03, 2.72366667e+03, 2.34466667e+03, 2.18833333e+03,
       2.51800000e+03, 2.73833333e+03, 2.07333333e+02, 1.33333333e+00])
{% endraw %} {% raw %} 
c2c.plot()
{% endraw %} {% raw %} 
archive_df.loc["EUV2001_093_08_35_28"]

path    /home/maye/uvis_archive/observations/EUV2001_0...
det                                                   EUV
Name: EUV2001_093_08_35_28, dtype: object
{% endraw %} {% raw %} 
p = obsdir / "index_repaired.tab"
{% endraw %} {% raw %} 
df = pd.read_csv(p, quotechar='"', skipinitialspace=True)

/home/maye/miniconda3/envs/py38/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3441: DtypeWarning: Columns (0,9,10,11,12,13) have mixed types.Specify dtype option on import or set low_memory=False.
  exec(code_obj, self.user_global_ns, self.user_ns)
{% endraw %} {% raw %} 
from planetarypy.pds.indexes import find_mixed_type_cols
{% endraw %} {% raw %} 
find_mixed_type_cols(df, fix=False)

9
1999-01-07 10:05:02.093
1999-01-07 10:08:34.0
N/A
EUV1999-01-07 17:05:02.000
Unnamed: 6
USTARE
to evaluate EUV and FUV functions.
7
32
0
0.1
0.2

['9',
 '1999-01-07 10:05:02.093',
 '1999-01-07 10:08:34.0',
 'N/A',
 'EUV1999-01-07 17:05:02.000',
 'Unnamed: 6',
 'USTARE',
 'to evaluate EUV and FUV functions.',
 '7',
 '32',
 '0',
 '0.1',
 '0.2']
{% endraw %} {% raw %} 
df.columns

Index(['9', '1999-01-07 10:05:02.093', '1999-01-07 10:08:34.0', 'EUV', 'N/A',
       'EUV1999-01-07 17:05:02.000', 'Unnamed: 6', 'USTARE',
       'to evaluate EUV and FUV functions.', '7', '32', '0', '0.1', '0.2',
       '0.3', '1', '0.4'],
      dtype='object')
{% endraw %} {% raw %} 
index.columns

Index(['FILE_NAME', 'OBSERVATION_TYPE', 'START_TIME', 'STOP_TIME',
       'TARGET_NAME', 'DATA_SET_ID', 'SPACECRAFT_CLOCK_START_COUNT',
       'SPACECRAFT_CLOCK_STOP_COUNT', 'INTEGRATION_DURATION',
       'COMPRESSION_TYPE', 'HI_VOLTAGE_POWER_SUPPLY_STATE',
       'OCCULTATION_PORT_STATE', 'SLIT_STATE', 'TEST_PULSE_STATE', 'ODC_ID',
       'RIGHT_ASCENSION', 'DECLINATION', 'SUB_SOLAR_LATITUDE',
       'SUB_SOLAR_LONGITUDE', 'SUB_SPACECRAFT_LATITUDE',
       'SUB_SPACECRAFT_LONGITUDE', 'PHASE_ANGLE', 'EMISSION_ANGLE',
       'SOLAR_INCIDENCE_ANGLE', 'CENTRAL_BODY_DISTANCE', 'DWELL_TIME',
       'H_LEVEL', 'D_LEVEL', 'filename'],
      dtype='object')
{% endraw %} {% raw %} 
index[index.filename.str.startswith("EUV")].iloc[0]

FILE_NAME                        /COUVIS_0001/DATA/D1999_007/EUV1999_007_17_05.LBL
OBSERVATION_TYPE                                                            USTARE
START_TIME                                              1999-01-07 17:05:01.949000
STOP_TIME                                               1999-01-07 17:08:37.949000
TARGET_NAME                                                                    NaN
DATA_SET_ID                                                  CO-J-UVIS-2-SPEC-V1.2
SPACECRAFT_CLOCK_START_COUNT                                      1/1294420183.000
SPACECRAFT_CLOCK_STOP_COUNT                                                    UNK
INTEGRATION_DURATION                                                           4.0
COMPRESSION_TYPE                                                              NONE
HI_VOLTAGE_POWER_SUPPLY_STATE                                                  OFF
OCCULTATION_PORT_STATE                                                      CLOSED
SLIT_STATE                                                         HIGH_RESOLUTION
TEST_PULSE_STATE                                                                ON
ODC_ID                                                                           7
RIGHT_ASCENSION                                                             -999.0
DECLINATION                                                                 -999.0
SUB_SOLAR_LATITUDE                                                          -999.0
SUB_SOLAR_LONGITUDE                                                         -999.0
SUB_SPACECRAFT_LATITUDE                                                     -999.0
SUB_SPACECRAFT_LONGITUDE                                                    -999.0
PHASE_ANGLE                                                                 -999.0
EMISSION_ANGLE                                                              -999.0
SOLAR_INCIDENCE_ANGLE                                                       -999.0
CENTRAL_BODY_DISTANCE                                                       -999.0
DWELL_TIME                                           1969-12-31 23:59:59.999999001
H_LEVEL                                                                        NaN
D_LEVEL                                                                        NaN
filename                                                         EUV1999_007_17_05
Name: 9, dtype: object
{% endraw %} {% raw %} 
obs.head()
filename slit nx ny nz int odcid x1 x2 y1 y2 name
0 EUV1999_016_19_47_15 2 1024 64 2 60 12 0 1023 0 63 alp vir
1 EUV1999_016_19_49_06 2 1024 1 123 1 13 0 1023 0 63 alp vir
2 EUV1999_016_20_08_10 2 1 64 399 1 11 0 1023 0 63 alp vir
3 EUV1999_016_20_16_14 2 1024 64 2 60 12 0 1023 0 63 alp vir
4 EUV1999_016_20_18_06 2 1024 1 123 1 13 0 1023 0 63 alp vir
{% endraw %} {% raw %} 
cols = ["index start_time stop_time detector target obsid_time unknown type comment "]
{% endraw %}